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A  series  of  Cockpit  Automation  Studies  are  being  performed  at  The  Naval  Air 
Development  Center  (NADC)  as  part  of  the  Adaptive  Function  Allocation  for 
Intelligent  Cockpits  program.  The  goal  of  the  program  is  to  develop  a  prospective 
set  of  human  performance  based  principles  and  guidelines  for  the  application  of 
adaptive  automation  technology.  These  guidelines  are  being  developed  for  the 
tactical  aircraft  cockpit,  but  may  be  suitable  for  other  applications.  NADC’s 
Cockpit  Automation  Studies  are  a  programmatic  series  of  studies  designed  to  take 
human  performance  theories  and  models  found  in  the  experimental  literature,  and 
extend  them  to  the  problems  of  adaptive  automation.  The  studies  will  serve  as 
the  basis  for  developing  an  environment  in  which  alternative  adaptive  automation 
concepts  may  be  demonstrated  and  validated. 

As  the  first  Cockpit  Automation  Study  (CAS),  the  baseline  study  served  to 
develop  a  basic  set  of  tasks  in  which  automation  concepts  could  later  be  applied. 
In  order  to  assess  the  effects  of  automation  on  human  performance,  it  is  necessary 
to  anticipate  what  the  effects  of  automation  are  likely  to  be  on  human 
performance,  so  that  the  experiment  tasks  developed  to  study  these  effects  will  be 
sensitive  to  the  anticipated  effects,  and  diagnostic  in  understanding  those  effects. 
Given  the  many  critical  tasks  found  in  an  aircraft  cockpit,  it  was  determined  that 
variations  of  two  common  laboratory  tasks  could  be  used  as  the  starting  point  in 
this  research.  One  task  was  a  two-dimensional,  first-order,  pursuit  tracking  task, 
which  is  theoretically  analogous  to  the  primary  flight  control  task  performed  by 
pilots.  The  second  task  is  a  derivative  of  the  binary  classification  task  used  in 
much  of  the  experimental  literature  examining  human  cognition.  This  'Tactical 
Assessment  Task”  (TAT)  requires  subjects  to  make  a  binary  decision 
(Friendly/Hostile)  to  targets  as  the^  move  across  a  screen.  Responses  for  the 
TAT  are  made  by  pressing  one  of  two  buttons  (Confirm/Designate)  when  the 
target  reaches  a  range  marker  on  the  display.  The  subject’s  response  time  and 
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accuracy  are  measured  to  ascertain  performance. 
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Various  aspects  of  the  two  tasks  used  in  this  study  were  manipulated  separately 
and  in  combination  to  determine  how  they  effect  performance.  It  was  desirable 
that  performance  on  each  of  the  individual  tasks  be  analyzed  and  interpreted 
separately  as  well  as  together  so  that  performance  trade-offs  between  tasks  could 
be  examined.  To  obtain  changes  in  performance,  the  driving  frequency  of  the 
tracking  task  was  manipulated,  while  the  number  of  target  types  and  target  event 
rate  were  manipulated  on  the  TAT.  All  these  manipulations  have  previously  been 
shown  to  effect  subject  performance,  and  may  be  operationally  used  to  manipulate 
task  difficulty.  However,  the  interpretation  of  results  when  such  tasks  are  placed 
in  combination  has  proven  difficult,  and  often  dependent  on  certain  theoretical 
assumptions.  Therefore,  one  of  the  objectives  of  the  baseline  study  was  to 
develop  a  methodology  which  would  allow  an  interpretation  of  "complex" 

(multiple  tasks,  performed  concurrently)  task  performance  with  a  minimal  of 
theoretical,  and  potentially  controversial,  assumptions. 


Several  iterations  of  the  baseline  experiment  were  performed  as  the  tasks  were 
developed.  Idiosyncrasies  in  the  operation  of  the  tasks  were  corrected  and  task 
difficulty  parameters  were  adjusted  to  obtain  sensitive  and  diagnostic  measures. 
The  data  collected  showed  that,  overall,  Root-Mean-Square  (RMS)  error  collected 
on  the  tracking  task  and  TAT  response  time  were  most  sensitive  to  manipulation 
of  difficulty  on  either  the  TAT  or  tracking  tasks.  TAT  percent  correct  was  found 
to  be  somewhat  less  diagnostic.  In  general,  manipulations  of  task  difficulty  had 
the  desired  effect  on  the  dependent  measures,  i.e.  increasing  the  difficulty  of  the 
task  and  decreased  performance.  The  data  from  the  final  iteration  of  the  study 
suggests  that  task  parameters  have  been  found  which  produce  results  that  may  be 
explained  in  the  context  of  an  optimal  arousal  hypothesis.  There  is  also  evidence 
which  suggests  that  performance  tradeoffs  were  made  in  the  complex  task  groups, 
and  that  beneficial  shifts  in  problem  solving  strategies  may  have  occurred  between 
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The  baseline  study  achieved  its  objectives  of:  1)  Developing  an  appropriate  task 
environment  for  examining  issues  in  adaptive  automation;  2)  Ensuring  that  the 
tasks  would  be  sensitive  and  diagnostic  to  changes  in  human  performance  when 
component  tasks,  or  aspects  of  those  tasks,  are  in  fact  automated;  3)  A 
methodology  was  found  allowing  the  potentially  complex  tradeoffs  to  be 
interpreted;  and  4)  Issues  and  caveats  for  future  research  were  identified.  Based 
on  the  results  of  this  study,  a  second  CAS  experiment  is  now  under  way  at  NADC 
and  a  third  is  being  developed.  Further,  the  task  has  served  to  elicit  a  number  of 
information  transfer  possibilities  for  6.3  programs. 
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INTRODUCTION 

The  demands  associated  with  flying  modern  tactical,  strategic  and  commercial 
aircraft  have  made  the  cockpit  a  prime  arena  for  the  development  of  technology 
designed  to  aid  human  operators.  The  development  of  ever  more  powerful 
computers,  expert  systems,  and  artificial  intelligence  technology  has  led  researchers 
and  system  designers  to  propose  that  decision  making  by  aircraft  pilots  may  be  aided 
dynamically  using  this  technology.  The  implementation  of  this  technology  may 
modify  the  tasks  normally  performed  by  pilots  in  any  of  a  number  of  ways,  in  order 
to  facilitate  the  best  performance  of  the  person-machine  system.  A  number  of 
programs  are  currently  underway  in  both  the  military  and  civilian  sectors  designed 
to  explore  the  question  of  what  is  possible  with  this  technology,  i.e.  "What  can  be 
done  to  change  the  pilot(s)  tasks?".  This  question,  however,  is  only  one  of  the 
questions  that  needs  to  be  addressed  before  this  technology  is  applied  to  the  aircraft 
cockpit. 

Perhaps  a  more  important  question  is  to  identify  the  consequences  and  problems 
likely  to  be  encountered  by  the  pilots/users  of  these  technologies,  i.e.  "What  should 
be  done  to  change  the  pilot(s)  tasks  given  his  capabilities?".  The  Adaptive  Function 
Allocation  for  Intelligent  Cockpits  program,  is  working  towards  answering  this  question 
by  identifying  human  performance  based  principles  and  guidelines  for  the  application 
of  adaptive  automation.  In  support  of  this  program,  a  series  of  Cockpit  Automation 
Studies  is  being  undertaken  at  NADC  to  generate  empirical  data  to  validate 
proposed  guidelines.  This  report  describes  the  results  of  the  first  Cockpit 
Automation  Study  (CAS). 

Background 

To  understand  the  potential  impact  of  adaptive  automation  on  human 
performance,  it  is  appropriate  tc  first  describe  adaptive  automation  and  how  it  might 
work.  Adaptive  automation  differs  from  conventional  or  nonadaptive  automation  in 
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that  the  tasks  performed  by  the  pilot  may  be  changed  in  some  way  by  the  automation 
system  itself.  The  automation  system  may  determine  when  and  how  a  task  is 
changed,  alter  which  tasks  are  performed  by  the  human  and  machine  components  of 
the  system,  it  may  take  on  certain  components  of  tasks  while  others  are  retained  by 
the  operator,  or  it  might  alter  the  operating  characteristics  of  specific  tasks  (Andes 
&  Rouse,  1990;  Parasuraman,  Bahri,  Deaton,  Morrison,  &  Barnes,  1990). 

Unlike  adaptive  systems,  conventional  automation  systems  may  change  the  way 
the  operator  performs  a  task  or  tasks,  but  the  operator  makes  the  determination  as 
to  when  the  system  is  activated  and,  quite  often,  what  form  it  will  take.  The  pitfalls 
of  this  strategy  are  that  it  fails  to  take  full  advantage  of  the  new  developments  in 
artificial  intelligence  and  expert  system  technology,  and  that  human  operators  are 
often  overloaded  at  those  times  when  such  technology  would  be  most  advantageous. 
Yet  it  is  precisely  these  times  when  human  operators  are  least  able  and  likely  to 
invoke  automation  (Parasuraman,  Bahri,  Deaton,  Morrison,  &  Barnes,  1990). 
Adaptive  automation,  however,  has  been  advocated  to  overcome  these  shortcomings. 
It  is  proposed  that  the  machine  component  of  an  adaptive  system  could  take 
responsibility  for  determining  when  it  is  appropriate  to  assume  responsibility  for 
performing  some  task,  or  changing  a  task’s  characteristics  so  that  it  is  more  easily 
performed.  The  exact  mechanisms  which  could  be  used  to  drive  such  a  complex 
decision  making  system  have  yet  to  be  clearly  defined  or  researched.  There  are, 
however,  several  alternatives  which  have  been  described. 

Table  1  represents  a  basic  outline  of  the  design  strategies  and  philosophies 
applied  by  designers  to  adaptive  automation  systems.  Each  strategy  reflects  a  general 
philosophy  regarding  what  should  drive  the  adaptive  allocation  algorithms:  mission 
requirements  or  operator  needs.  Currently,  designers  implement  decisions  based 
upon  their  understanding  of  human  capabilities,  technology,  and  their  individual 
design  preferences.  The  strategies  described  in  Table  1  include:  1)  Allocation,  in 
which  well  defined  tasks  are  reallocated  between  the  human  and  aiding  system;  2) 
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Partitioning,  in  which  aspects  of  a  task  (or  a  sub-task)  are  performed  by  the  human 
operator  while  other  aspects  of  the  task  are  performed  by  the  aiding  system  (different 
aspects  of  the  task  may  also  be  performed  by  the  aid  system  and  human  operator  at 
different  times);  and  3)  Transformation,  in  which  operator  interaction  with  a  task 
is  essentially  changed  by  the  aiding  system  in  order  to  change  the  requirements  on 
the  human  operator.  The  particular  strategy  and  philosophy  chosen  by  designers  for 
adaptive  aiding  systems  is  typically  dependent  on  the  criticality  of  the  tasks  to 
successful  completion  of  the  mission  and,  to  a  large  degree,  technology  limitations. 
However,  experience  with  existing  aiding  systems  has  shown  that  the  operator’s 
capabilities  and  limitations  must  also  play  a  critical  role  in  the  success  of  the  aiding 
system  in  meeting  its  design  requirements. 

The  implementation  of  adaptive  automation  could  have  a  number  of  impacts  on 
pilot  performance,  both  beneficial  and  detrimental,  because  of  the  changing  set  of 
tasks  and  performance  demands  with  which  human  operators  will  be  forced  to  cope 
as  a  result  of  adaptive  automation.  Before  such  systems  can  be  implemented,  the 
impact  on  task  performance  of  this  new  technology  must  be  fully  understood 
(Parasuraman,  Bahri,  Deaton,  Morrison,  &  Barnes,  1990). 

A  number  of  benefits  have  been  anticipated  from  the  development  of  adaptive 
automation  systems.  First,  adaptive  automation  could  alleviate  some  of  the  negative 
aspects  of  nonadaptive  automation  such  as  performance  decrements  associated  with 
long  term  monitoring,  loss  of  situation  awareness,  and  manual  skill  degradation 
(Parasuraman,  Bahri,  Deaton,  Morrison,  and  Barnes,  1990).  This  has  been  suggested 
for  several  reasons.  An  operator  with  an  adaptively  automated  system  would  actually 
perform  monitoring  tasks  some  of  the  time.  Therefore,  the  number  and  duration  of 
periods  in  which  they  would  be  involved  in  system  monitoring  should  be  diminished, 
preventing  decrements  in  performance  commonly  associated  with  vigilance.  The 
reduced  potential  for  a  vigilance  decrement  should  have  a  positive  effect  on  overall 
performance.  Related  to  this  issue  is  the  tendency  of  human  operators  to  be  less 
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aware  of  the  "status"  of  a  task  when  not  actively  involved  in  task  performance. 
Adaptive  automation  could  increase  the  operator’s  awareness  of  the  task  because,  in 
fact,  they  would  perform  it  some  of  the  time.  Similarly,  manual  control  skills  should 
be  less  degraded  in  an  adaptively  automated  system  because  the  operator  is  forced 
to  exercise  manual  control  whenever  the  system  is  not  automated.  (For  a  review  of 
these  issues  see  Hancock,  Chignell,  &  Lowenthal,  1985;  Noah  &  Halpin,  1986; 
Parasuraman  &  Bowers,  1987;  Rouse,  1976,  1988;  Wickens  &  Kramer,  1985). 

The  potential  problems  with  adaptive  automation  have  been  pointed  out  by  a 
number  of  human  performance  researchers.  Weiner  (1988),  in  his  field  studies  of 
pilot  problems  encountered  with  current  technology  commercial  aircraft,  has  noted 
that  one  new  task  that  pilots  will  need  to  cope  with  is  intervention  when  an 
automated  system  fails.  Deficiencies  during  the  performance  of  intervention  could 
have  catastrophic  consequences.  As  noted  by  Logan  (1990)  and  Gluckman  (1990), 
the  time  required  for  operators  to  adjust  to  the  dramatic  changes  in  task  load,  such 
as  those  associated  with  automated  system  failure  can  vary  greatly,  and  has  not  been 
completely  explored.  Yet,  this  one  factor  could  render  adaptive  automation  useless 
in  many  settings,  including  aviation,  if  the  temporal  adjustment  of  attention  resources 
is  sufficiently  slow. 

CAS-1:  Baseline  Study 

The  present  paper  represents  the  beginning  of  a  programmatic  line  of  research 
into  the  interactive  effects  of  cockpit  tasks  on  human  and  system  performance.  Since 
this  is  a  relatively  new  area  of  research,  a  number  of  basic  issues  must  be  addressed 
regarding  methodology.  Therefore,  some  of  the  main  objectives  of  this  study  include: 

1)  Providing  baseline  empirical  data  for  predicting  performance  in  future  research; 

2)  Developing  appropriate  data  interpretation  procedures;  and  3)  Assessing  the 
sensitivity  and  appropriateness  of  the  dependent  measures  and  independent 
manipulations.  The  ultimate  result  of  this  effort  will  be  baseline  data  for  human 
performance  which  will  be  used  as  a  basis  for  assessing  the  effects  of  adaptive 
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automation  in  later  studies.  As  the  CAS  research  program  matures,  the  data 
collected  across  studies  will  allow  the  costs  and  benefits  of  alternative  adaptive 
automation  strategies  to  be  evaluated  relative  to  performance  without  such  strategies. 
With  these  data  in  hand,  it  will  be  possible  to  develop  a  set  of  human  performance 
based  guidelines  and  principles  for  the  application  of  adaptive  automation  systems 
in  the  tactical  aircraft  environment. 

In  order  to  meet  the  goals  of  the  Cockpit  Automation  Program,  NADC’s 
reconfigurable  cockpit  (RC)  is  being  utilized  as  the  primary  research  platform.  This 
system  incorporates  multiple  display  and  logic  processors  to  control  multiple 
independent  tasks,  or  an  integrated  flight  simulation.  The  relatively  distributed 
processing  architecture  of  the  RC  allows  various  tasks  to  be  implemented  at  different 
levels  of  complexity  and  realism  with  minimal  impact  on  the  operation  of  other  tasks 
in  the  scenario.  In  addition,  the  inherent  flexibility  of  this  system  makes  it  an 
appropriate  candidate  for  an  evolutionary  series  of  studies  because  of  the  degree  to 
which  the  developed  code  for  running  and  analyzing  the  tasks  may  be  reutilized  as 
tasks  are  inserted  into,  and  removed  from  a  common  testing  environment.  Thus,  the 
RC  provides  the  ability  to  manipulate  the  specific  set  of  tasks  which  are  presented 
at  any  given  time,  thereby  enabling  tasks  to  be  systematically  isolated,  and  combined, 
as  well  as  automated  or  released  from  automation.  Further,  and  of  particular 
importance  to  the  study  of  adaptive  automation,  all  of  the  mentioned  changes  in  task 
parameters  can  be  systematically  made  without  subject  or  task  interruption. 

A  preliminary  analysis  of  the  tasks  performed  in  tactical  fighter  aircraft  served  as 
the  basis  for  developing  appropriate  tasks  for  this  investigation  (Cohen,  1990).  Tasks 
were  chosen  and  developed  based  upon  their  ecological  validity,  (i.e.  their 
generalizability  to  flight  activities),  the  ability  to  operationally  define  levels  of  task 
difficulty,  and  the  degree  to  which  they  could  be  related  to  standard  human 
performance  tasks  found  in  the  research  literature.  Two  basic  tasks  were  developed 
for  the  present  study  on  the  basis  of  this  analysis.  One  task  was  chosen  to  represent 
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a  continuous  perceptual/motor  task,  with  requirements  similar  to  those  found  in 
controlling  aircraft  flight.  The  actual  task  developed  was  a  first-order,  two- 
dimensional,  pursuit  tracking  task.  The  second  task  was  chosen  to  represent  a 
visual/cognitive  task,  analogous  to  monitoring  a  horizontal  situation  display  (HSD). 
Thus,  the  Tactical  Assessment  Task  (TAT)  developed  for  this  study  required 
monitoring  the  movement  of  a  number  of  targets  which  would  then  be  identified  by 
the  subject  as  they  traversed  the  screen.  Both  the  tracking  and  TAT  tasks  have  well 
documented  analogs  in  the  human  performance  literature  (Briggs  &  Blaha,  1969; 
Briggs  &  Johnsen  1973;  Briggs  &  Shinar,  1972;  Poulton,  1974;  Sternberg,  1967; 
Wickens,  1984;).  As  a  consequence,  there  is  considerable  theoretical  basis  for 
interpreting  the  data  generated  from  these  tasks. 

In  addition  to  identifying  generic  tasks  present  in  advanced  tactical  aircraft,  the 
cockpit  analysis  also  suggested  that  a  key  aspect  of  tactical  missions  was  shifts  in  the 
difficulty  and/or  complexity  of  flight  tasks  which  occur  during  the  course  of  a 
mission.  The  present  study  was  therefore  designed  to  provide  information  about 
performance  on  the  tracking  and  TAT  tasks  at  a  variety  of  levels  of  task  difficulty 
and  complexity.  This  information  will  also  be  valuable  for  developing  appropriate 
tasks  to  study  the  effects  of  adaptive  automation.  As  discussed  earlier,  it  may  be 
desirable  to  invoke  an  adaptive  automation  system  when  tasks  increase  in  difficulty 
since  during  these  periods  of  time  automation  may  be  most  beneficial.  Thus, 
obtaining  baseline  performance  data  on  tasks  which  increase  in  difficulty  without 
automation  will  provide  important  comparative  data  for  assessing  potential  benefits 
of  automation  during  these  instances. 

A  number  of  performance  theories  have  been  proposed  in  the  literature  which 
describe  how  the  concurrent  performance  of  multiple  tasks  effect  each  other  and 
overall  performance.  For  instance,  resource  theory  provides  a  convenient  and  useful 
tool  for  predicting  possible  outcomes  for  the  manipulations  utilized  in  this  study 
(Kahnemann,  1973;  Navon  &  Gopher,  1979;  Wickens,  1984).  According  to  this 
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theory,  task  performance  is  directly  related  to  the  ability  of  the  operator  to  allocate 
available  processing  resources  to  cope  with  current  task  demands.  As  long  as  these 
demands  do  not  exceed  processing  capability,  increases  in  task  demand  will  not  effect 
performance,  i.e.  there  will  be  no  reduction  in  performance.  Instead  of  a  reduction 
in  performance  efficiency,  resource  theory  suggests  that  more  resources  are  deployed, 
or  existing  resources  are  used  more  efficiently.  Once  all  available  resources  are 
dedicated  to  performing  a  task,  any  further  increase  in  the  difficulty  of  a  task  will 
result  in  a  corresponding  decrease  in  task  performance.  At  these  levels  of  task 
difficulty,  differences  between  tasks,  and  the  effects  of  adding  or  taking  away  tasks 
can  be  clearly  seen.  If  this  version  of  resource  theory  is  accepted,  it  becomes 
apparent  that  one  additional  goal  of  the  baseline  study  must  be  to  ensure  that  the 
tasks  used  do  not  overload  subjects  to  such  a  degree  that  uniformly  poor 
performance  is  found,  or  underload  the  subjects  to  a  degree  that  no  changes  are  seen 
in  subject  performance.  Either  result  would  make  the  interpretation  of  the  results 
problematic. 

The  issue  of  task  underload  or  overload  becomes  even  more  critical  in  the 
context  of  automation.  The  anticipated  result  in  applying  automation  is  that,  in 
effect,  a  task  is  no  longer  being  performed  by  the  human  operator.  If  the  size  of  this 
benefit  is  going  to  be  measured  in  order  to  assess  to  what  degree  operator  resources 
are  released  for  application  to  other  tasks,  it  is  necessary  that  the  remaining  (non- 
automated)  tasks  still  impose  an  appropriate  level  of  workload  such  that  the 
magnitude  of  change  is  measurable.  Further,  if  a  complex  task  (e.g.  flying  an 
airplane),  consists  of  a  variety  of  component  tasks,  and  then  the  effects  from 
automating  various  combinations  of  those  component  tasks  are  going  to  be 
compared,  it  must  be  determined  if  those  tasks  combinations  are  likely  to  be  using 
similar  levels  of  resources.  In  other  words,  that  the  removing  of  a  component  task 
through  automation  does  not  so  underload  the  operator  that  the  resulting 
performance  measures  do  not  reflect  the  subject’s  effort  in  performing  the  task. 
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The  notion  of  underload/overload  has  theoretical  importance  in  r.ddition  to  the 
practical  issues  of  data  collection.  These  concepts  are  embodied  within  arousal 
theory  (Broadbent,  1971;  Easterbrook,  1959;  Hockey,  1976;  Yerkes  &  Dodson,  1908). 

According  to  this  explanation  of  human  performance,  performance  efficiency  is  best 
when  the  level  of  stimulation  associated  with  a  task  is  neither  too  little  nor  too  great, 
i.e.,  an  optimal  level  of  stimulation  is  achieved.  This  theory  could  be  applied  to 
understanding  human  performance  in  adaptively  automated  systems.  It  is  possible 
that  automation  of  system  functions  should  sufficiently  reduce  task  stimulation  that 
a  situation  of  underarousal  could  arise,  and  would  therefore  result  in  a  decrease  in 
task  performance.  Equally  likely  is  the  possibility  for  manual  task  demands  to  create 
a  level  of  overarousal  which  would  then  result  in  a  decrease  in  task  performance. 
From  this  perspective,  it  would  be  important  for  an  adaptively  automated  system  to 
maintain  a  balance  between  task  difficulty  and  automation  such  that  a  state  of 
equilibrium  at  the  optimal  level  of  task  stimulation  is  maintained.  Although  this 
position  is  very  appealing,  it  has  one  major  drawback  for  predicting  the  outcome  of 
the  present  study.  Since  this  is  a  baseline  study  no  a  priori  knowledge  is  known 
about  levels  of  task  stimulation  which  will  be  produced  by  the  task  utilized.  Thus, 
it  is  impossible  to  determine  where  the  tasks  are  positioned  on  the  arousal  curve,  and 
interpretation  of  the  results  from  this  perspective  can  only  be  made  post-hoc. 

Although  the  arousal  position  does  not  allow  predictions  to  be  drawn  about  the 
outcome  of  the  present  study,  resource  theory  does  provide  some  direction.  Once 
appropriate  levels  of  simple  task  performance  are  found  for  the  tracking  and  TAT 
tasks,  it  can  be  predicted  that: 

(1)  Performance  on  any  task  by  itself  would  be  better  than  when  the  task  was 
performed  in  conjunction  with  a  second  task; 

(2)  A  direct  relationship  between  performance  and  task  difficulty/complexity 
would  be  found  such  that  performance  on  the  easy  tasks  was  always  greater 
than  performance  on  the  hard  tasks. 
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(3)  When  both  tasks  were  present,  the  best  overall  performance  would  occur  in 
the  combination  containing  both  easy  versions  of  the  tasks  followed  by  pairs 
in  which  only  one  of  the  tasks  in  the  pair  was  a  hard  task.  The  worst 
performance  was  expected  in  the  task-pairing  in  which  both  tasks  were  hard 
tasks. 
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SUBJECTS 

A  total  of  six  Naval  Air  Development  Center  employees  were  used  across  three 
iterations  of  the  baseline  study.  The  results  from  the  first  two  iterations  served  to 
debug  the  experiment  apparatus,  and  optimize  the  task  characteristics  and  subject 
protocol.  Only  the  results  of  the  final  iteration  will  be  discussed  here.  In  this  study 
one  woman  and  two  men  served  as  subjects.  All  three  subjects  had  normal,  or 
corrected  to  normal,  vision.  One  of  the  men  was  left  handed  while  the  remaining 
subjects  were  right  handed.  None  of  the  subjects  had  any  known  physical  anomaly 
that  would  impact  task  performance. 


APPARATUS 

The  experiment  was  conducted  using  the  NADC  Reconfigurable  Cockpit  (RC). 
The  RC  is  a  mid-fidelity,  single  pilot,  fixed-base,  flight  simulator.  The  system  was 
originally  designed  to  develop  and  validate  glass-cockpit  displays  and  controls. 
However,  for  the  baseline  study  its  capabilities  were  modified  to  serve  as  a  flexible 
testing  environment  for  human  performance  research.  The  current  experiment 
required  the  development  of  a  two-dimensional  tracking  task  and  a  decision  making 
task  (also  referred  to  as  the  Tactical  Assessment  Task  or  TAT).  Both  tasks  were 
presented  on  6"  Cathode  Ray  Tube  (CRT)  displays  which  were  part  of  Multi- 
Function  Displays  (MFDs)  embedded  in  the  RC  cockpit.  Subject  responses  to  the 
TAT  were  made  by  depressing  MFD  buttons  while  the  tracking  task  was  controlled 
through  a  conventional  flight  control  stick.  Although  the  RC  was  used  to  run  and 
coordinate  the  experiment,  limitations  in  its  original  architecture  dictated  that  an 
external  Zenith  286  computer  system  incorporating  a  Scientific  Solutions  LabTender 
card  would  be  required  to  provide  millisecond  timing  resolution  for  subjects’ 
responses  to  the  TAT. 
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The  tactical  assessment  task  was  displayed  on  a  Barco  CM-22  video  monitor 
located  in  the  upper  center  quadrant  of  the  display  panel  (See  Figure  1).  The  TAT 
display  format  consisted  of  an  own-ship  reference  surrounded  by  two  range  rings  (See 
Figure  2).  Targets  appeared  in  the  top  half  of  the  display  at  a  predetermined  rate 
throughout  each  experiment  session.  Targets  moved  down,  and  to  some  degree 
across,  the  display  at  a  fixed  rate  such  that  all  targets  passed  inrough  the  inner  and 
outer  range  markers,  and  then  off  the  monitor.  Subjects  were  to  classify  the  targets 
as  Enemy  or  Friendly  as  soon  as  they  crossed  the  outer  range  marker.  Twenty 
percent  of  the  targets  were  defined  so  that  they  appeared  either  touching  or  just 
inside  the  outer  range  ring,  and  therefore  required  immediate  responses  from  the 
subject.  These  "pop-up"  targets  had  the  effect  of  making  the  TAT  somewhat  less 
predictable,  and  were  intended  to  mimic  the  behavior  of  radar  displays  encountering 
low  observable  aircraft,  electronic  jamming,  or  low  level  flight.  From  a  theoretical 
viewpoint,  the  "pop-up"  targets  were  added  to  ensure  that  subjects  continually 
monitored  the  TAT  task  by  reducing  the  predictability  of  signal  events.  With  the 
addition  of  this  type  of  target,  subjects  could  not  anticipate  the  occurrence  of  the 
next  signal  given  the  apparent  distance  and  speed  of  oncoming  targets.  Thus, 
subjects  were  less  able  to  take  task  contingent  "time  outs"  from  monitoring.  The  lack 
of  such  time-outs  has  been  shown  to  increase  the  overall  difficulty  of  monitoring 
tasks  (Jerison  and  Pickett,  1964;  Richter,  Senter,  and  Warm,  1981;  Scerbo,  Warm, 
and  Fisk,  1987).  The  dependent  measures  used  for  the  TAT  task  were  percent 
correct  and  response  time.  Both  of  these  measures  were  utilized  because  it  is  not 
possible  to  interpret  response  time  or  accuracy  without  assessing  if  subjects  have 
shifted  their  performance  criterion  between  them,  i.e.  there  has  been  a  analog  of  a 
speed-accuracy  tradeoff  (Briggs  &  Shinar,  1972;  Pachella,  1974). 

The  tracking  task  utilized  was  a  first-order,  two-dimensional  compensatory  task. 
The  task  was  displayed  through  a  Sony  PVM-8221  composite  video  monitor  mounted 
in  the  upper  right  comer  of  the  cockpit  (See  Figure  1).  The  task  display  format 
consisted  of  a  red  colored  "target"  symbol  and  a  white  colored  "own-ship"  symbol,  as 


(7)  TAT  (7)  Flight  Control  Stick 

(?)  Tracking  task  (?)  Data  Entry  Keyboard  Input 

(?)  Throttles  (?)  Seat 


Figure  1.  NADCs  Reconfigurable  Crewsiation  Simulator. 
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(T)  Ownship 

(2)  Outer  Range  Ring  (As  targets  approach  they  must  be  identified) 

(3)  Target  (to  be  identified) 

(5)  'TARGET*  Prompt-  signals  that  a  target  must  be  identified 
(?)  Target  which  has  been  correctly  identified 
(?)  Target  which  will  need  to  be  identified 

(7)  Response  buttons 

(?)  Target  which  has  been  missed  or  incorrectly  identified 


Figure  2.  Tactical  Assessment  Task  (TAT)  Display  Format. 
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shown  in  Figure  3.  Both  symbols  were  approximately  1  by  0.5  cm  in  size.  The 
tracking  task  display  also  contained  a  ground  reference  symbol  and  a  set  of  altitude 
scales  to  assist  the  subject  in  judging  target  movements.  The  ground  symbol  moved 
appropriately  in  response  to  the  subject’s  control  inputs,  while  the  altitude  scales 
remained  fixed.  Both  of  these  aids  were  presented  in  green,  and  care  was  taken  to 
ensure  that  both  the  target  plane  and  own-ship  could  not  bisect  these  lines.  As  with 
many  aircraft  displays,  the  self-reference  point,  or  own-ship,  remained  as  a  fixed 
point  in  the  center  of  the  monitor.  The  dependent  measure  used  for  this  task  was 
Root  Mean  Square  Error  (RMSE)5. 

PROCEDURE 

Several  days  in  advance  of  the  first  experiment  sessions,  subjects  were  given  a 
briefing  package.  The  package  contained  a  thorough  explanation  of  the  experiment, 
a  verbal  description  of  the  tracking  task  and  the  TAT  task,  as  well  as  copies  of 
Figures  1,  2,  and  3.  Subjects  were  also  instructed  to  memorize  the  symbol  set 
presented  in  Figure  4  so  that  they  could  identify  each  symbol  with  100%  accuracy. 
Self-test  practice  sheets  were  provided  to  facilitate  this  process.  Prior  to  the  start  of 
each  experimental  session,  a  short  test  of  the  symbols  was  administered  to  the 
subjects  to  ensure  that  correct  identification  was  possible.  Copies  of  these  materials 
may  be  found  in  Appendix  1. 

In  order  to  ensure  that  the  initiation  of  subject’s  responses  were  consistent,  the 
word  'TARGET'  would  appear  at  the  upper  center  of  the  TAT  display  when  a  target 


’The  procedure  used  in  collecting  data  involved  storing  actual  horizontal  and  vertical 
deviations  in  pixels  at  a  rate  of  10  Hz.  During  post-processing,  these  data  were  then 
converted  to  RMSE  through  the  use  of  the  formula  (Poulton,  1974): 

RMSE  = 

where  X  =  Ordinate,  Y  =  Abscissa,  n  =  Number  of  Samples. 


n 
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(7)  Ownship  bug  (fixed  to  display) 

(2)  Target  (to  be  tracked) 

(3)  Ground  representation 
(J)  Altitude  scale 

Figure  3.  Tracking  Task  Display  Format. 
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i&jrg.  4,  Symbols  Used  in  the  Tactical  Assessment  Task. 
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became  eligible  for  a  response.  Timing  started  when  the  TARGET  symbol 
appeared.  Only  one  target  was  eligible  for  response  at  any  given  time.  Responses 
were  made  by  pressing  one  of  two  MFD  buttons  surrounding  the  TAT.  The  upper 
left  button  was  labeled  CON  and  was  used  to  CONfirm  friendly  targets.  The  upper 
right  button  was  labeled  DES  and  was  used  to  DESignate  enemy  targets  (See  Figure 
2).  Subjects  had  3  seconds  from  the  time  the  'TARGET'  symbol  appeared  to  make 
a  detection  response.  Responses  occurring  after  the  3  second  interval  were 
considered  false  alarms,  and  were  eliminated  from  the  analyses.  At  all  times, 
subjects  responded  to  TAT  targets  using  the  index  finger  of  their  left  hand.  Between 
TAT  events,  subjects  rested  their  hand  on  a  fixed  position  on  the  left  armrest.  Mean 
distance  to  the  response  buttons  on  the  TAT  MFD  was  57  cm. 

The  difficulty  of  the  TAT  was  manipulated  in  two  different  ways.  First,  the  rate 
of  critical  events  was  changed.  The  low  event  rate  (slow  condition)  had  an  average 
inter-stimulus  interval  (ISI)  of  15  seconds  with  a  range  of  12  -  18  seconds,  while  the 
ISI  for  the  high  event  rate  (fast  condition)  was  10  seconds  with  a  range  from  7  -  13 
seconds.  TAT  difficulty  was  also  varied  by  changing  the  number  of  symbol  types 
presented.  This  manipulation  was  conceptually  analogous  to  changing  the  number 
of  items  which  had  to  be  remembered,  i.e.  memory  set-size.  In  the  easy  condition, 
only  two  of  the  possible  eight  symbols,  the  diamond  and  circle,  were  used,  (See 
Figure  4).  All  eight  symbols  seen  in  Figure  4  were  used  in  the  hard  condition. 

With  regard  to  the  tracking  task,  subjects  were  instructed  to  track  the  target  to 
the  best  of  their  ability  by  manipulating  the  force  stick  with  their  right  hand  only. 
Perfect  tracking  was  defined  as  keeping  the  own-ship  reference  centered  on  the 
target  at  all  times.  Two  target  driving  frequencies,  the  rate  of  direction  changes, 
were  used  to  create  two  levels  of  difficulty  in  the  tracking  task  (easy  and  hard).  The 
easy  tracking  condition  was  defined  as  having  a  driving  frequency  of  0.061  Hz,  while 
the  hard  condition  was  defined  as  having  a  driving  frequency  twice  that  of  the  easy 
(driving  frequency  =  0.121  Hz).  A  central,  floor-mounted  force  stick  was  used  for 
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control  inputs  to  the  tracking  task,  and  was  located  as  in  a  conventional  cockpit 
configuration  (See  Figure  1).  The  force  requirements  to  operate  the  flight  control 
stick  on  the  simulator  were  similar  to  those  of  an  F-14  in  normal  flight. 

The  factorial  combination  of  the  tracking  and  TAT  tasks  at  each  level  of  task 
difficulty  created  a  total  of  14  conditions,  as  can  be  seen  in  Table  2.  Sue  of  these 
groups  consisted  of  "simple"  tasks  in  which  subjects  were  required  to  perform  either 
the  tracking  task  or  the  TAT.  The  remaining  eight  conditions  contained  "complex” 
tasks  in  which  subjects  were  required  to  perform  both  the  tracking  and  TAT  tasks 
simultaneously.  Each  subject  participated  in  all  of  the  experiment  conditions  creating 
a  completely  within-factors  design.  In  order  to  avoid  problems  with  fatigue,  subjects 
participated  in  three  1  to  1  1/2  hour  experiment  sessions  distributed  over  three 
consecutive  days.  The  First  session  consisted  of  a  ten-minute  review  of  the  subject 
briefing  material,  followed  by  three  five-minute  blocks  of  training  (1  block  simple 
tracking,  1  block  simple  TAT,  1  block  complex  task  consisting  of  both  tracking  and 
TAT).  All  training  was  done  with  the  easy  task  difficulty  levels  (Tracking  -  easy, 
TAT  -  slow  event  rate  and  two  symbol  types).  The  training  was  then  followed  by 
four  10-minute  experiment  blocks.  Sessions  two  and  three  began  with  two  minutes 
of  practice  on  each  of  the  three  training  task  combinations  used  in  session  1  (in  a 
randomly  selected  order)  followed  by  five  10-minute  experiment  blocks.  Conditions 
were  randomly  assigned  to  experiment  blocks,  with  the  constraint  that  some  simple 
and  complex  task  conditions  had  to  occur  each  day.  Experiment  blocks  within  each 
session  were  separated  by  a  five-minute  rest  period. 


Complex  Task  Condition  Simple  Task  Condition 
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While  the  experiment  design  would  allow  an  analysis  of  variance  (ANOVA)  to 
be  performed,  the  limited  number  of  subjects  used  in  the  final  iteration  of  this  study 
does  not  create  sufficient  data  to  justify  its  use.  Therefore,  the  data  are  presented, 
summarized  and  discussed  without  the  use  of  formal  statistical  tests.  Hence,  all  of 
the  results  and  discussion  must  be  considered  tentative  until  replicated  in  later  CAS 
research.  The  data  pertaining  to  each  of  the  dependent  measures  will  be  presented 
in  both  graphic  and  tabular  form.  Standard  errors  of  the  means  will  be  presented 
in  both  formats  so  that  an  informal  evaluation  of  effect  size  and  significance  can  be 
made. 


The  most  straightforward  approach  to  assessing  the  complex  effects  on 
performance  is  to  evaluate  the  effects  on  simple  task  combinations  (only  the  tracking 
or  the  TAT  task)  independently  from  those  of  complex  task  combinations  (both  the 
tracking  and  the  TAT  tasks).  Once  these  have  been  outlined,  differences  between 
performance  on  the  simple  vs.  complex  tasks  will  be  discussed. 


Simple  Task  Results 

Tracking  Task.  Mean  RMSE  for  all  subjects  in  all  conditions  are  presented  in 
Table  3,  and  are  graphically  portrayed  in  Figure  5.  As  can  be  seen  in  Figure  5, 
RMSE  was  lower  for  the  easy  as  compared  to  the  hard  condition  in  the  simple  task 
combination.  This  effect  was  fairly  robust  as  can  be  seen  by  the  non-overlapping 
standard  error  bars  for  these  conditions. 

Tactical  Assessment  Task  (Reaction  Time).  Mean  of  median  reaction  times  for 
all  task  combinations  are  presented  in  Table  4,  and  in  Figure  6.  One  should  note 
that  the  reaction  time  data  was  more  variable  relative  to  the  effect  sizes  than  either 
the  RMSE  or,  as  will  be  seen  below,  the  percent  correct  data.  Therefore,  the  results 
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Tabic  3.  Mean  RMS  Error  for  Each  of  the  Experiment 
Conditions. 
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Table  4.  Mean  of  Median  Reaction  Times  in  Seconds  for 
Each  of  the  Experiment  Conditions 


NOTE:  NUMBERS  IN  PARENTHESIS  ARE  THE  STANDARD  ERROR  OF  THE  MEAN. 
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based  upon  this  measure  are  less  conclusive.  It  would  appear,  however,  that  there 
is  a  main  effect  for  event  rate  in  the  simple  task  data.  Reaction  times  for  the  slow 
event  rate  conditions  were  faster  than  those  to  the  fast  event  rate  conditions.  The 
data  do  not  show  an  effect  for  symbol  types  used,  nor  does  there  appear  to  be  an 
interaction  between  event  rate  and  symbol  type. 

Tactical  Assessment  Task  (Percent  Correct!  Mean  percent  correct  data  are 
presented  in  Table  5  and  Figure  7.  Perhaps  the  most  immediately  noticeable  result 
in  both  the  table  and  the  figure  is  the  very  high  level  of  performance  for  all  of  the 
task  conditions.  Equally  evident  is  the  very  low  variability  of  these  data.  Together, 
these  results  suggest  that  performance  changes  may  have  been  minimized  due  to  a 
ceiling  effect.  Despite  this  problem,  several  interesting  effects  emerged  as  can  be 
seen  in  Figure  7.  W  h  egard  to  the  simple  task  groups,  no  overall  differences  for 
event  rate,  or  n'  i r’  jt  of  symbol  types  is  present.  There  does,  however,  appear  to 
be  an  interac  uon  between  event  rate  and  number  of  symbols.  With  reference  to  the 
conditions  in  which  8  symbol-types  were  used,  no  difference  in  percent  correct 
between  the  slow  and  fast  event  rates  is  present.  However,  in  the  2  symbol-type 
groups,  a  higher  percent  correct  was  achieved  in  the  slow  event  rate  condition  than 
the  other  three  conditions,  and  a  lower  percent  correct  is  present  in  the  fast  event 
rate  condition  than  is  present  in  the  other  conditions. 

Complex  Task  Results 

Because  the  complex  task  conditions  involve  performance  on  both  the  tracking 
and  the  TAT  tasks,  the  discussion  of  performance  effects  in  these  conditions  will  be 
discussed  in  terms  of  dependent  measures  as  opposed  to  task  type  (tracking  or  TAT). 

RMSE.  As  can  be  seen  in  Figure  5,  in  all  of  the  complex  task  conditions  RMSE 
was  lower  for  the  conditions  in  which  the  TAT  was  coupled  with  the  easy  tracking 
task  than  those  in  which  it  was  paired  with  the  hard  tracking  task.  There  also  is  a 
tendency  for  task  conditions  in  which  the  fast  TAT  event  rate  was  performed  in 
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conjunction  with  the  tracking  task  to  have  a  lower  RMSE  than  conditions  in  which 
the  slow  TAT  event  rate  was  utilized.  No  differences  emerged  as  a  consequence  of 
symbol  types  used,  and  no  interactions  between  any  of  the  three  factors  (tracking 
difficulty,  TAT  event  rate,  and  TAT  symbol  types)  were  present. 

Reaction  Time.  Illustrated  in  Figure  6  are  the  effects  on  reaction  time  for  the 
complex  task  groups.  Most  noticeable  in  the  figure  is  that  reaction  times  to  signals 
in  the  TAT  were  faster  when  this  task  was  coupled  with  the  hard  tracking  task  than 
when  it  was  coupled  with  the  slow  tracking  task.  No  other  main  effects  or 
interactions  between  TAT  event  rate,  symbol  types,  or  tracking  difficulty  are  evident 
in  the  figure. 

Percent  Correct.  Percent  correct  data  for  the  complex  task  groups  is  presented 
in  Figure  7.  Once  again,  interpretation  of  these  data  may  be  difficult  since  ceiling 
effects  may  be  present.  With  this  in  mind,  it  appears  as  though  there  are  no  main 
effects  for  tracking  difficulty,  TAT  event  rate,  or  symbol  types.  There  does,  however, 
appear  to  be  a  subtle  three-way  interaction  between  these  factors.  With  regard  to 
the  combinations  of  tracking  and  TAT  in  which  only  2  symbol  types  were  used,  a 
greater  number  of  signals  were  identified  in  the  conditions  in  which  the  hard  tracking 
task  was  present  than  those  in  which  the  slow  tracking  task  was  used.  This  effect  was 
not  modified  by  event  rate.  This  overall  pattern  of  results  does  not  appear  in  the 
conditions  in  which  all  eight  symbol  types  were  used.  In  this  case,  the  effects  brought 
about  by  the  increase  in  tracking  difficulty  were  moderated  by  event  rate.  In  the 
slow  event  rate  conditions,  performance  when  the  task  was  coupled  with  the  easy 
tracking  task  exceeded  that  of  the  condition  in  which  the  hard  tracking  task  was  used. 
The  reverse  occurred  when  the  tracking  tasks  were  combined  with  the  fast  event  rate 
TAT. 
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Performance  Differences  Between  Simple  and  Complex  Tasks 

The  comparisons  that  follow  are  important  for  their  potential  implications  for 
automation.  They  will  be  re-examined  in  greater  detail  as  part  of  the  discussion 
section. 

RMSE.  With  regard  to  tracking  task  performance  (Figure  5)  within  each  level 
of  tracking  task  difficulty  RMSE  was  always  greater  for  the  complex  as  compared  to 
the  simple  task  conditions.  However,  overall  RMSE  for  the  complex  conditions  in 
which  the  easy  tracking  task  was  utilized  were  not  higher  than  those  found  for  the 
(simple)  hard  tracking  task  performed  alone. 

Reaction  Time.  As  was  found  with  RMSE,  it  can  be  seen  in  Figure  6  that 
overall,  reaction  times  were  longer  in  the  complex  task  conditions  than  they  were  in 
the  simple  task  conditions.  Also  evident  is  a  tendency  for  the  simple  task  reaction 
times  with  the  fast  event  rate  to  be  similar  to  those  of  the  complex  task  conditions 
in  which  the  TAT  was  coupled  with  the  hard  tracking  task. 

Percent  Correct.  No  differences  between  simple  and  complex  task  performance, 
were  found  for  accuracy  (Figure  7).  Once  again,  a  lack  of  differentiation  between 
conditions  with  regard  to  percent  correct  may  reflect  a  ceiling  effect. 

Pop-up  vs.  Normal  Targets 

As  can  be  seen  in  Table  6,  slight  differences  between  pop-up  and  normal  targets 
were  found  on  either  measure  of  performance  (percent  correct  and  reaction  time). 
The  changes  were  in  the  anticipated  direction,  i.e.  increased  reaction  times  and  RMS 
error  for  pop-ups  relative  to  normal  targets.  Further,  similar  results  were  found  in 
the  complex  and  simple  task  conditions  and  there  were  no  interactions  between 
target  type  (pop-up  or  normal)  and  the  simple  or  complex  tasks  were  found. 
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The  present  study  is  the  first  in  a  series  of  cockpit  automation  studies  (CAS) 
addressing  human  performance  issues  in  adaptive  automation  as  it  applies  to  the 
tactical  aircraft  cockpit.  Since  this  effort  represents  the  first  in  a  line  of 
programmatic  studies,  this  experiment  was  designed  to  accomplish  the  following 
goals:  (1)  Adapt  NADC’s  Reconfigurable  Crewstation  (RC)  to  implement  the  tasks 
to  be  used  in  the  CAS  program,  and  collect  appropriate  human  performance  data. 
(2)  Develop  a  set  of  research  tasks  analogous  to  those  found  in  a  tactical  aircraft 
cockpit  which  are  also  comparable  to  standard  laboratory  tasks  used  to  study  human 
performance.  (3)  Collect  and  analyze  preliminary  performance  data  on  several 
versions  of  the  baseline  tasks  in  order  to  ensure  that  empirically  different  levels  of 
task  difficulty  were  developed.  (4)  Develop  an  initial  understanding  of  how  multiple 
cockpit  tasks  impact  performance  on  each  other,  as  well  as  how  performance  of  the 
tasks  in  combination  relate  to  performance  of  the  tasks  when  performed  alone. 
These  data  will  be  used  to  determine  the  effects  on  human  performance  due  to 
automation  and  release  from  automation  which  will  be  evaluated  in  later  studies.  (5) 
Evaluate  the  performance  metrics  used  to  ensure  that  they  reflect  meaningful 
differences  in  performance  and  are  sufficient  for  understanding  performance  on 
cockpit  tasks.  The  results  of  the  CAS-1  study  suggest  that  almost  all  of  the  goals 
have  been  attained.  Each  of  these  goals  will  be  discussed  in  turn  in  the  remainder 
of  the  discussion  as  well  as  suggestions  for  alternative  procedures  and  performance 
indices  when  appropriate. 

Appropriateness  of  the  RC  to  Human  Performance  Research 

The  RC  was  successfully  modified  to  allow  appropriate  research  tasks  to  be 
developed.  Several  major  alterations,  including  an  external  timing  mechanism  to 
monitor  subjects’  responses  to  the  Tactical  Assessment  Task  (TAT)  were  required. 
However,  it  was  possible  to  obtain  accurate  and  reliable  data.  In  addition  to  the 
immediate  goal  of  task  development  for  the  CAS-1  experiment,  the  RC  was  also 
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identified  as  a  highly  flexible  and  powerful  research  vehicle  that  can  be  further 
adapted  to  accommodate  future  goals  of  the  CAS  program. 

Task  Development:  Implementation  and  Levels  of  Task  Difficulty 

A  first-order  tracking  task  and  a  tactical  assessment  task  (TAT)  have  been 
successfully  implemented  on  the  RC.  After  several  iterations  in  developing  these 
tasks,  manipulations  of  driving  frequency,  event  rate,  and  number  of  symbols  were 
used  to  alter  task  difficulty.  The  results  of  the  experiment  suggest  that  the 
manipulations  of  task  difficulty  or  complexity  in  the  simple  task  conditions  were,  for 
the  most  part  successful,  and  generally  support  a  resource  view  of  human 
performance.  With  regard  to  the  manipulation  of  driving  frequency  in  the  tracking 
task,  the  data  suggest  that  doubling  the  driving  frequency  significantly  reduces 
tracking  task  performance  (increases  RMS  error).  This  suggests  that  subjects  utilize 
more  resources  when  performing  the  tracking  task  at  high  driving  frequencies  than 
at  low  frequencies.  With  regard  to  the  TAT,  the  results  suggest  that  increasing  the 
event  rate  increases  subjects’  reaction  times  but  does  not  effect  their  detection 
accuracy.  According  to  resource  theory,  such  a  result  suggests  there  is  a  direct 
relationship  between  the  amount  of  resources  required  and  the  number  of  events  that 
must  be  processed.  Such  a  result  is  not  uncommon,  and  has  been  found  using  other 
tasks  which  feature  discrete  events  (Gluckman,  1990).  No  effects  for  TAT 
performance  were  found  for  the  manipulation  of  symbol  set  size,  and  no  differences 
in  detection  accuracy  were  found  for  any  of  the  manipulations  of  the  TAT. 

The  overall  lack  of  sensitivity  of  the  percent  correct  measure,  and  the  general 
lack  of  differentiation  between  the  two  symbol  set  sizes  were  not  anticipated.  Upon 
review  of  the  training  and  experiment  procedures,  however,  several  explanations  for 
these  results  were  found.  First,  subjects  were  trained  to  one  hundred  percent 
proficiency  for  symbol  identification  prior  to  the  experiment.  Second,  because  a 
sufficiently  long  period  of  time  was  given  for  subjects  to  respond  to  signals  (3 
seconds),  few  were  missed,  and  subjects  could  afford  to  take  time  to  ensure  high 
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accuracy.  Lastly,  the  memory  set  sizes  that  were  utilized  in  the  present  study  (2  or 
8  symbols),  did  not  exceed  the  limits  of  working  memory  (7  +_  2  items)  (Miller, 
1967).  Therefore,  once  a  target  became  active,  it  could  readily  be  compared  against 
the  symbol  representations  in  memory. 

The  last  manipulation  that  was  utilized  in  this  experiment  was  to  make  target 
occurrences  more  uncertain  by  adding  pop-up  targets.  As  would  be  anticipated,  both 
percent  correct  and  reaction  times  to  these  stimuli  tended  to  be  degraded  relative  to 
the  normal  stimuli.  Although  the  results  are  not  compelling,  the  theoretical  and 
ecological  rationale  for  incorporating  pop-up  targets  is  important  and  should  be 
investigated  further  in  future  CAS  studies. 

In  addition  to  their  theoretical  importance,  the  results  of  the  simple  task 
performance  data  have  some  interesting  ecological  implications  which  provide  some 
direction  for  future  research.  It  is  possible  with  current  technology,  that  the  number 
of  active  targets  that  might  appear  on  a  Radar  Intercept  Officer’s  (RIO)  radar  screen 
might  be  greater  or  less  than  the  event  rates  utilized  in  this  study.  It  is  less  likely, 
however,  that  the  number  of  different  symbols  which  would  be  used  to  designate  both 
friendly  and  unfriendly  targets  would  exceed  8  (the  number  used  in  the  present 
experiment).  Moreover,  RIO  training  procedures  would  be  at  least  as  stringent  as 
those  used  in  this  experiment  and  would,  therefore,  ensure  that  few  mistakes  in 
identification  would  occur.  These  ecological  factors  suggest  that  future  modifications 
on  the  TAT  should  focus  on  changes  in  the  event  rate  rather  than  the  complexity  of 
the  symbol  set.  It  is  possible  that  further  increases  in  event  rate  will  effect  both 
speed  and  accuracy  rather  than  only  a  shift  in  the  speed  with  which  subjects  can 
identify  targets.  This  might  occur  because  more  targets  will  need  to  be  identified, 
the  relative  amount  of  time  to  identify  targets  will  be  reduced,  and  subjects  will  be 
afforded  less  time  to  consider  their  response  choice. 
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Complex  Task  Performance:  Task  Interaction  and  Comparison  to  Simple  Tasks. 

Several  important  findings  are  present  in  the  complex  task  data  which  address 
goal  number  4;  understanding  complex  task  performance  and  how  it  differs  from 
simple  task  performance.  This  goal  is  critical  to  the  CAS  program  because  the 
complex  task  conditions  are  more  representative  of  real  cockpit  activities.  The 
results  of  the  manipulations  of  task  difficulty  in  the  complex  task  groups  were 
generally  consistent  with  those  of  the  simple  task  groups.  As  with  simple  task 
performance,  the  manipulation  of  driving  frequency  in  the  tracking  task  had  the  most 
pronounced  effect  on  task  performance.  Also  consistent  with  the  simple  task  results, 
the  number  of  symbol  types  used  had  no  effect  on  performance  as  shown  by  either 
TAT  dependent  measure  (percent  correct  and  reaction  time). 

Unlike  the  simple  task  results,  no  main  effect  for  event  rate  emerged  in  the 
performance  data  for  the  complex  task  groups  on  either  dependent  measure. 
However,  a  trend  for  a  three  way  interaction  of  event  rate,  driving  frequency,  and 
number  of  symbols  was  present  in  the  percent  correct  data  only.  This  result  was  not 
anticipated  and  cannot  be  accommodated  by  either  a  resource  or  arousal  explanation. 
One  should  note  that  the  effect  size  for  this  three-way  interaction  is  very  small,  and 
from  an  operational  perspective  represents  a  difference  in  detection  accuracy  of  only 
one  or  two  signals  in  any  condition.  Therefore,  it  is  likely  that  this  interaction  is  a 
result  of  a  methodological  constraint  as  opposed  to  differences  in  resource  drain  or 
task  stimulation  as  would  be  suggested  by  the  resource  or  arousal  models. 

Although  the  percent  correct  data  did  not  provide  much  insight  into  complex  task 
performance,  several  important  results  emerged  in  the  reaction  time  and  RMS  error 
data.  The  pattern  of  effects  that  manipulating  tracking  task  difficulty  had  on 
performance  was  remarkably  consistent  both  within  the  complex  task  conditions  and 
between  the  simple  and  complex  conditions.  Overall,  both  RMS  error  and  reaction 
times  were  elevated  by  the  need  to  engage  in  both  the  TAT  and  tracking  tasks 
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relative  to  only  having  to  perform  one  of  these  tasks.  This  result  is  consistent  with 
a  resource  view  of  human  performance.  In  this  case,  sufficient  amounts  of  additional 
resources  may  have  been  necessary  to  perform  both  tasks  in  the  complex  task 
conditions  that  performance  was  reduced  relative  to  simple  task  performance.  The 
specific  nature  of  the  decrease  in  performance  in  the  complex  task  groups  is  quite 
interesting.  If  one  looks  at  the  results  from  the  perspective  of  changes  produced  by 
altering  the  tracking  task  difficulty  a  consistent  pattern  emerges  between  the  reaction 
time  and  RMS  error  measures.  When  the  tracking  task  is  easy,  RMSE  is  low  and 
reaction  time  is  slow.  When  tracking  difficulty  is  hard,  RMSE  is  high  but  reaction 
time  is  fast. 

From  a  resource  theory  perspective,  one  might  not  find  a  trade-off  in 
performance  efficiency,  such  as  the  one  described  above,  to  be  unusual.  In  fact, 
these  have  been  explicitly  described  in  resource  theory  as  a  performance  operating 
characteristic  (POC)  (Wickens,  1984).  A  POC  represents  the  optimal  level  of 
performance  on  a  set  of  tasks  that  must  be  processed  concurrently.  At  each  end  of 
the  POC  curve  is  the  level  of  performance  that  would  be  attained  if  only  one  of  the 
tasks  had  to  be  performed.  Along  the  function  connecting  these  endpoints  are 
performance  levels  when  both  tasks  are  performed  concurrently.  The  process  of 
constructing  a  POC  curve  for  any  given  set  of  tasks  normally  involves  systematically 
instructing  subjects  to  focus  differing  amounts  of  resources  or  attention  (from  0  to 
100  percent)  to  the  performance  of  one  of  the  two  tasks.  Using  this  procedure  a 
sample  set  of  points  along  the  POC  can  be  derived  which  describes  the  POC  for  that 
particular  set  of  tasks.  The  data  from  the  present  study  could  be  mapped  into  the 
framework  of  a  POC  if  one  assumes  that  the  performance  shifts  brought  about  by  the 
change  in  tracking  task  difficulty  resulted  in  a  shift  in  subjects’  focus  of  attention  or 
resource  allocation  strategy.  If  this  is  the  case,  it  would  have  to  be  dependent  upon 
some  factor  internal  to  the  subjects  since  subjects  were  instructed  to  perform  both 
tasks  equally  in  all  of  the  complex  task  conditions. 
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There  remains  the  problem  of  why  the  tasks  flip-flop  according  to  the  difficulty 
of  the  tracking  task,  and  not  the  event  rate  of  the  TAT.  In  addition,  one  might 
wonder  why  the  prominent  effect  of  event  rate  present  in  the  simple  TAT 
performance  was  not  present  in  the  complex  task  conditions.  One  answer  to  these 
questions  might  be  provided  if  one  speculates  about  how  subjects  perceived  the 
difficulty  of  the  tasks.  One  factor  which  might  influence  how  subjects  perceive  the 
tracking  and  TAT  could  be  the  amount  of  performance  feedback  provided  by  each 
task.  Tracking  task  feedback  is  continuous  because  subjects  may  assess  their 
performance  on  the  task  any  time,  as  often  as  they  like.  When  the  own-ship 
reference  is  overlapping  the  target  then  the  feedback  is  positive,  otherwise,  it  is 
negative.  In  either  case,  the  feedback  alerts  subjects  to  their  performance  and 
subjects  are  continuously  afforded  the  opportunity  to  correct  their  actions.  In  the 
TAT,  feedback  for  attending  to  the  task  is  both  positive  and  negative,  however  it  can 
only  occur  with  the  appearance  of  a  new  target,  or  with  a  change  in  the  status  of  a 
target  to  one  which  requires  a  response.  No  opportunity  is  provided  to  alter 
responses  on  the  TAT.  Because  of  these  differences,  subjects  may  have  shifted  their 
attention  resources  toward  the  tracking  task  because  it  provided  them  with  a  greater 
degree  of  positive  feedback,  particularly  when  the  tracking  task  was  easy.  When  the 
tracking  task  was  hard,  and  the  amount  of  time-on-target  was  reduced,  the  tracking 
task  would  be  less  reinforcing  and  therefore,  relatively  more  positive  feedback  may 
have  been  provided  by  the  TAT.  Thus,  subjects  shifted  their  attention  toward  the 
TAT.  With  this  explanation,  the  event  rate  manipulation  may  have  had  no  effect 
because  sufficient  levels  of  resources  where  either  drawn  away  from,  or  added  to, 
performance  of  the  TAT  so  that  the  effect  of  this  manipulation  was  minimized.  The 
precise  reason  for  both  the  shift  in  performance  due  to  the  change  in  tracking  task 
difficulty  and  the  lack  of  an  event  rate  effect  in  the  complex  task  conditions  cannot 
be  assessed  in  this  study. 

Although  overall  differences  between  simple  and  complex  task  performance 
emerged,  complex  task  performance  was  not  always  worse  than  simple  task 
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performance,  as  was  predicted  based  on  the  resource  model.  Upon  careful 
inspection  of  the  results  one  might  note  that  RMS  error  for  the  hard  tracking  task 
alone  was  not  different  from  RMS  error  on  the  easy  tracking  task  when  it  was 
performed  in  conjunction  with  any  of  the  TAT  combinations.  Moreover,  reaction 
times  to  the  fast  event  rate  TAT  conditions,  when  performed  alone,  did  not  appear 
to  differ  from  reaction  times  to  the  slow  event  rate  TAT  when  combined  with  the 
hard  tracking  task.  Once  again,  the  catalyst  for  this  effect  may  be  the  driving 
frequency  of  the  tracking  task.  Event  rate  and  symbol  set  size  can  be  eliminated  as 
potential  causes  since  no  significant  differences  in  complex  task  performance  were 
found  as  a  result  of  changes  in  these  task  manipulations. 

A  possible  explanation  for  the  equality  of  simple  and  complex  task  performance 
under  certain  conditions  hinges  around  the  potential  for  differences  in  problem 
solving  strategies  which  have  been  implicated  with  regard  to  tracking  tasks.  Poulton 
(1974)  and  Wickens  (1984)  have  noted  that  at  certain  levels  of  tracking  difficulty, 
subjects  change  their  tracking  strategy  from  one  that  minimizes  position  errors  at  the 
expense  of  velocity  errors  to  one  that  minimizes  velocity  errors  at  the  expense  of 
position  errors.  At  low  driving  frequencies,  subjects  may  try  to  precisely  follow  the 
motions  of  the  target  resulting  in  overcompensation;  the  velocity  oriented  approach. 
However,  at  higher  frequencies,  they  adopt  a  strategy  in  which  high  frequency 
changes  are  filtered  out  and  lower  frequency  changes,  or  general  directional  changes 
are  favored;  the  position  oriented  approach.  In  essence,  they  smooth  out  their 
motions  to  follow  more  gross,  obvious  changes  in  the  direction  or  speed  of  the  target. 
The  effect  of  this  change,  in  terms  of  the  dependent  measure  RMSE,  can  be  a 
reduction  in  overall  error  under  high  frequency  changes  since  less  over-  or  under¬ 
compensation  for  target  motion  are  made. 

One  might  assume,  that  using  the  strategy  which  minimizes  position  errors, 
resulting  in  a  more  smooth  flow,  would  require  fewer  resources  than  one  centered 
on  minimizing  velocity  errors  since  subjects  would  not  have  to  sample  for  changes  in 
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direction  or  velocity  as  often,  and  fewer  control  corrections  would  be  required.  The 
present  results  may  be  due  to  subjects’  shifting  their  tracking  strategy  in  the  complex 
task  conditions.  This  shift  of  focus  or  strategy  might  interact  with  their  overall 
performance  and  the  relative  amounts  of  positive  feedback  they  perceive  as  coming 
from  the  tracking  and  TAT  tasks.  Using  the  current  measurement  techniques,  there 
is  no  way  to  determine  which  strategy  was  used  by  subjects  in  any  of  the  conditions 
used  in  the  experiment  or  if  changes  in  strategy  were  prompted  by  certain 
combinations  of  tasks.  This  information  could  only  be  provided  by  adding  a 
measure  of  control  stick  movements;  e.g.  control  reversals  in  the  X  and  Y  axes. 
When  combined  with  the  information  about  the  dynamics  driving  both  the  target  and 
the  output,  a  clear  time-dependent  picture  of  subjects’  problem  solving  strategy  could 
be  gained. 

The  fact  that  performance  on  any  of  the  complex  tasks  was  equivalent  to  that  of 
simple  task  performance  challenges  one  of  the  basic  assumptions  at  the  heart  of  most 
automation  concepts.  Specifically,  it  is  assumed  that  as  the  number  of  tasks  that 
must  be  performed  decreases,  performance  increases.  Automation  of  tasks, 
therefore,  should  increase  performance.  The  results  of  this  study,  although  they 
represent  only  baseline  information  about  non-automated  tasks,  suggest  that  this  is 
not  always  the  case.  As  a  result,  automation  strategies  in  which  fixed  levels  of 
automation  are  provided,  (as  is  found  in  conventional  automated  systems),  will  not 
always  provide  performance  benefits.  The  results  do  suggest  that  the  concept  of 
adaptive  automation  may,  however,  be  viable  since  one  of  its  key  features  is  to 
moderate  task  load  as  suggested  in  Table  1. 

Given  the  limited  scope  of  the  present  results  and  their  generalizability  to  real 
flight  missions,  the  human  performance  data  from  this  study  suggest  that  if  pilot 
consent  is  a  critical  aspect  of  the  adaptive  automation  system,  then  the  task  that  is 
perceived  as  most  difficult  by  the  pilot  might  be  the  best  candidate  for  automation 
because  it  is  these  tasks  that  pilots  are  least  inclined  to  perform.  In  addition,  the 
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hard  task  is  the  one  which  they  may  be  least  involved  with,  and  therefore,  pilots 
might  be  more  inclined  to  release  the  difficult  task  to  automation. 

The  data  obtained  in  this  study  could  have  implications  for  the  application  of 
partitioning  and  transformation  strategies.  It  may  be  that  the  automation  system 
should  apply  a  strategy  that  maintains  equilibrium  in  the  amount  of  demand  each 
task  places  on  the  pilot  so  that  no  particular  task  is  favored  and  pilots  are  equally 
involved  in  all  tasks.  In  this  way,  pilots  would  remain  engaged  with  all  tasks 
appropriately.  Thus,  through  the  appropriate  use  of  adaptive  automation,  situational 
awareness  could  be  maintained  while  overall  levels  of  task  load  are  reduced  to 
manageable  levels. 

These  conclusions  are  highly  speculative  and  very  limited  in  scope.  Many 
additional  factors  including  motivation,  mission  requirements,  and  technological  limits 
will  influence  how  the  empirical  results  generated  in  the  CAS  program  can  actually 
be  applied. 

Evaluation  of  Measurement  Procedures. 

The  last  goal  of  this  investigation  was  to  validate  the  measurement 
instrumentation  and  procedures  used  in  the  RC  to  assess  performance.  All  of  the 
measures  taken  for  this  study  proved  appropriate  and,  for  the  most  part,  provided 
diagnostic  information  about  subjects’  performance  and  behavior.  There  were, 
however,  several  shortcomings  which  should  be  addressed  in  future  CAS  studies. 
Clearly,  less  information  was  gained  about  problem  solving  strategies  subjects  used 
than  is  necessary.  Although  RMSE  provided  an  index  of  overall  performance, 
additional  information  about  accuracy,  problem  solving  strategies,  and  perhaps 
resource  utilization  for  the  tracking  task  could  have  been  provided  by  an  analysis  of 
control  stick  control  reversals.  The  incorporation  of  these  data  is  within  the  current 
capabilities  of  the  RC  experiment  platform,  and  would  provide  valuable  information 
to  help  explore  the  possibility  that  subjects  adopt  a  smoothing  strategy  under  certain 
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circumstances.  From  an  ecological  standpoint  Wickens  (1984)  notes  that  such 
strategies  are  directly  applicable  to  primary  flight  activities.  Moreover,  adopting  the 
incorrect  strategy  during  any  particular  phase  of  flight  could  have  disastrous 
consequences.  This  realization  provides  additional  justification  for  the  addition  of 
more  advanced  tracking  task  measurement  techniques. 

A  second  and  much  more  complex  shortcoming  of  the  current  measurement 
techniques  is  that  they  do  not  provide  an  index  of  overall  problem  solving  strategy 
or  a  single  metric  of  complex  task  performance.  At  the  present  time,  little  direction 
about  how  to  solve  this  problem  exists  in  the  literature.  One  place  to  start,  however, 
would  be  to  look  at  time-line  analyses  of  performance  and  incorporate  a  more 
detailed  post-test  debriefing  that  focuses  on  questions  related  to  problem  solving 
strategies.  Lastly,  an  issue  that  has  been  raised  in  the  automation  literature,  and  has 
been  suggested  as  a  means  for  invoking  automation  is  workload.  Future  CAS 
experiments  might  incorporate  workload  measurements  in  order  to  assess  this  issue. 

Implications  for  Future  Studies. 

The  complex,  and  often  subtle  effects  of  the  interactions  among  the  TAT  and 
tracking  task  manipulations  suggests  that  performance  in  such  a  complex  task 
environment  can  not  be  readily  predicted  from  single  task  performance,  nor  even 
similar  complex  tasks.  The  interaction  effects  appear  to  be  due  to  the  perception  of 
the  task(s)  by  the  subject.  The  perception  of  task  difficulty  cause  the  subject  to  make 
changes  in  the  resources  made  available  to  a  complex  task,  and/or  the  shifting  of  a 
resource  allocation  strategy.  The  introduction  of  automation  will  likely  complicate 
these  effects  because  automation  will  further  change  the  nature  of  the  complex  task 
(e.g.  through  the  transformation  of  a  task).  The  impact  of  automation  may  therefore 
be  negative,  e.g.  under  conditions  where  the  subject  is  inclined  to  shift  to  a  less 
optimal  performance  strategy.  Such  an  effect  would  be  undesirable  for  the 
application  of  adaptive  automation,  and  unexpected  on  an  intuitive  basis.  Future 
studies  looking  at  the  effects  of  automation  on  human  performance  will  therefore 
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have  to  establish  weir  thought  out  control  conditions  so  that  the  impact  on  task 
performance  can  be  more  clearly  understood. 

In  order  to  understand  performance  on  tasks  with  a  continuous  control  (tracking) 
component,  future  studies  should  collect  and  analyze  data  regarding  control  inputs, 
as  well  as  RMSE  reflecting  control  performance.  The  potential  shift  of  continuous 
control  strategies,  e.g.  smoothing  the  control  inputs,  would  have  the  net  effect  of 
making  the  tracking  task  easier.  This  strategy  must  be  detected  and  taken  into 
account  if  strategy  tradeoffs  in  the  context  of  a  more  complex  task  are  to  be 
understood. 

Given  the  complex  interrelationship  of  performance  on  the  tasks  employed  in  this 
study,  there  may  be  less  transfer  from  this  task  to  actual  cockpit  task  performance 
than  was  at  first  anticipated.  Little  data  are  available  to  indicate  what  level  of  task 
fidelity  is  necessary  to  accurately  predict  performance  from  relatively  low  fidelity 
simulation  tasks  to  the  highly  complex  and  dynamic  aircraft  piloting  task.  Therefore, 
future  studies  in  the  program  must  continue  to  develop  increasingly  complex  and 
dynamic  tasks,  and  compare  the  results  obtained  in  them  to  both  conventional 
laboratory  tasks  and  the  real-world  task.  Only  in  performing  such  a  series  of  studies 
will  it  be  possible  to  ascertain  what  level  of  fidelity  in  task  performance  is  necessary 
to  predict  performance  in  the  context  of  adaptive  automation.  This  issue  will 
certainly  remain  a  part  of  the  CAS  program  in  the  coming  years. 

Future  research  will  also  continue  to  increase  the  levels  of  complexity  of  the  tasks 
used.  Another  CAS  experiment  is  currently  underway  to  assess  the  impact  of  adding 
a  communication  task  to  the  array  of  tasks  (TAT  and  tracking  tasks)  used  in  this 
study.  Further,  a  basic  automation  algorithm  is  being  implemented  with  these  tasks 
to  determine  how  automation  of  the  component  tasks  impacts  complex  task 
performance.  The  second  CAS  study  will  be  followed  by  similar  studies  utilizing 
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alternative,  and  more  complex  adaptive  automation  strategies  such  as  those  described 
in  Table  1. 


NADC-91 028-60 


CONCLUSION 


44 


The  present  study  was  designed  to  serve  as  the  basis  for  future  investigations  into 
the  effects  of  adaptive  automation  on  pilot  performance  in  an  advanced  tactical 
aircraft.  As  the  first  in  a  series  of  studies,  the  baseline  study  had  a  number  of 
significant  goals.  A  primary  goal  was  to  develop  an  appropriate  research  platform 
for  conducting  programmatic  research  into  the  human  performance  issues  of  adaptive 
automation.  Consistent  with  this  goal  was  the  desire  to  study  adaptive  automation 
in  the  context  of  tasks  representative  of  both  traditional  laboratory  tasks  found  in  the 
human  performance  literature  and  at  the  same  time  meaningful  to  the  prediction  of 
pilot  performance  in  the  tactical  aircraft  cockpit.  Further,  the  baseline  study  was 
designed  to  allow  several  levels  of  task  difficulty  to  be  created  on  the  tasks 
developed.  Finally,  the  study  was  intended  to  demonstrate  a  methodology  for 
examining  and  interpreting  data  under  conditions  in  which  multiple  tasks  and 
dependent  measures  are  utilized. 

The  results  of  the  study  indicate  that  by  and  large  the  goals  have  been  met.  The 
Reconfigurable  Cockpit  (RC)  has  been  chosen  as  the  basis  for  conducting  CAS 
research  at  NADC.  It  has  proven  to  be  a  flexible  and  powerful  research  platform. 
The  tasks  currently  implemented  on  the  RC  are  representative  of  both  critical  tasks 
found  in  a  tactical  aircraft,  and  traditional  laboratory  tasks.  A  two-dimensional,  first 
order  pursuit  tracking  task  served  as  a  perceptual-motor  task  and  was  representative 
of  the  primary  flight  control  task  found  in  tactical  aircraft.  A  Tactical  Assessment 
Task  (TAT)  decision  making  task  was  developed  from  the  laboratory  binary 
classification  task  and  was  representative  of  a  situational  awareness  task  found  in 
monitoring  a  tactical  radar  display.  As  a  result  of  the  baseline  study,  it  was  possible 
to  assess  the  sensitivity  of  the  dependent  measures  utilized,  and  the  effectiveness  of 
the  independent  manipulations  in  both  complex  task  and  single  task  performance. 
The  present  study  also  served  as  a  vehicle  for  developing  approaches  and  models 
which  will  allow  for  the  interpretation  of  task  interactions  independent  of  effects  due 
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to  the  introduction  of  automation.  Further,  aspects  of  these  tasks  may  be  used  to 
demonstrate  the  effects  of  alternative  adaptive  automation  systems  on  human 
performance.  A  methodology  has  been  developed  for  interpreting  the  results,  and 
generally  meaningful  results  were  obtained.  The  results  from  the  baseline  study 
demonstrated  that  the  tasks  will  be  sensitive  and  diagnostic  for  the  study  of 
automation  when  it  is  introduced  to  this  environment. 

After  considerable  work  with  the  Reconfigurable  Crew-station  (RC),  it  was 
possible  to  collect  tracking  data,  response  time  data  and  percent  correct  data  with 
adequate  resolution  for  the  assessment  of  human  performance  in  a  complex  task 
environment  with  resemblance  to  the  operational  task  environment  of  a  tactical 
aircraft.  Two  lines  of  future  research  have  been  defined  based  on  the  results  of  the 
baseline  study.  One  line  of  research  will  use  the  tasks  developed  in  this  study  in 
addition  to  a  communication  task  which  is  currently  being  implemented  on  the  RC. 
An  additional  line  of  research  will  develop  a  more  realistic,  higher  fidelity  derivative 
of  the  TAT  and  tracking  task  which  is  more  appropriate  for  the  use  of  skilled  pilots 
and  will  be  more  representative  of  the  tactical  aircraft  cockpit.  The  studies  planned 
in  the  context  of  these  two  lines  of  research  will  incorporate  aspects  of  current 
adaptive  automation  systems  so  that  their  impact  on  human  performance  may  be 
empirically  studied.  The  research  will  allow  an  examination  of  operator  (pilot) 
strategies  in  performing  a  variety  of  complex  tasks.  Further,  comparing  across  the 
two  lines  of  research  will  permit  the  relative  merits  of  predicting  meaningful 
performance  for  the  tactical  aircraft  pilot  based  on  data  collected  in  basic  research. 
The  ability  to  discuss  these  issues  at  the  end  of  the  Adaptive  Function  Allocation  for 
Intelligent  Cockpits  program  will  represent  a  significant  and  meaningful  contribution 
to  the  human  performance  literature  and  the  application  of  adaptive  automation 
technology  to  the  tactical  aircraft. 
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SUBJECT:  Adaptive  Automation  Pilot  Study:  Task  description  and  subject 
briefing. 

Thank  you  for  agreeing  to  participate  in  the  adaptive  automation  study.  This 
experiment  is  one  of  a  series  of  experiments  being  performed  to  assess  how  different 
types  of  automation  which  may  be  used  in  an  aircraft  cockpit  affect  pilot 
performance.  Our  goal  is  to  try  and  bridge  the  gap  between  the  findings  of 
traditional  laboratory  experiments  so  that  they  impose  the  same  kinds  of  demands 
as  would  be  experienced  by  a  Tactical  Aircraft  pilot.  By  doing  this,  we  hope  to  learn 
how  each  of  the  tasks  affects  the  ability  to  perform  other  tasks  when  they  are  being 
performed  concurrently.  Hie  findings  will  then  be  used  to  help  determine  what  tasks 
should  be  automated  and  when.  This  will  ensure  that  the  pilot’s  resources  are 
allocated  most  effectively. 

In  this  experiment  you  will  perform  two  tasks:  a  tracking  task  and  a  tactical 
assessment  task. 

Tracking  Task.  You  might  want  to  think  of  this  task  as  taking  the  place  of  flying 
an  airplane.  Your  task  is  to  fly  the  airplane  so  that  you  keep  a  set  of  crosshairs 
centered  on  the  target  airplane  at  ail  times. 

In  effect,  the  crosshairs  are  painted  on  your  windshield.  If  the  target  is 
above  you,  you  must  pull  the  stick  back,  thus  pu  ling  the  nose  of  your 
airplane  up  to  get  closer  to  the  target.  If  the  target  is  below  you,  you 
must  push  the  nose  down  by  pushing  the  stick  forward.  To  move  to  the 
left,  push  the  stick  to  the  left.  To  move  the  nose  right,  move  the  stick  to 
the  right.  Remember,  you  should  try  to  track  the  target  as  accurately  as 
possible  at  all  times. 

The  wavy  line  you  will  see  at  the  bottom  of  the  screen  represents  the 
ground.  It  is  there  as  a  visual  reference  only  -  you  do  not  need  to  worry 
about  flying  into  it 


Tactical  Assessment  Task.  This  task  requires  that  you  monitor  a  series  of  targets 
which  are  moving  across  the  screen  and  identify  them  as  Enemy  or  Friendly  when 
they  come  into  range.  In  effect,  the  display  is  similar  to  what  you’d  see 
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monitoring  a  radar  display  of  oncoming  aircraft.  You  must  identify  the  target 
that  is  near  the  outer  semi-circle,  or  is  between  the  inner  and  outer  semi-circle. 


Only  one  target  will  need  to  be  identified  at  a  time.  When  the  computer 
detects  an  eligible  target  you  will  see  the  word  TARGET  appear  at  the  top 
of  the  screen.  When  you  see  the  TARGET  prompt,  you  should  use  your 
left  index  finger  to  push  either  the  CONfirm  or  DESignate  button  as 
quickly  and  accurately  as  possible.  Friendly  targets  are  CONfirmed,  while 
hostile  targets  are  DESignated. 

When  you  are  not  identifying  a  target,  it  is  very  important  that  you  keep 
vour  left  index  finger  on  the  red  dot  (located  on  the  left  armrest).  (This 
is  because  we  are  timing  how  long  it  takes  you  to  press  the  correct  button, 
and  therefore  for  the  time  to  mean  anything,  you  must  always  start  your 
hand  from  the  same  point.)  Likewise,  try  not  to  start  moving  your  hand 
before  the  TARGET  prompt  appears  on  the  screen.  You  may  monitor  the 
progress  of  various  symbols  as  they  move  across  the  screen,  howeve-  you 
should  not  begin  your  response  until  the  word  TARGET  appears. 

NOTE:  Not  all  the  symbols  will  be  seen  before  they  cross  into  the  area 
in  which  they  must  be  identified.  Some  will  "pop-up"  between  or  near  the 
two  semi-circles. 

-You  only  have  a  few  seconds  in  which  to  identify  the  targets,  so  you  must  pay 
attention  to  both  the  TAT  and  TRACKING  tasks.  Both  are  equally  important. 

-  Please  be  careful  not  to  touch  any  of  the  controls,  pedals,  or  switches.  We  have  not 
turned  off  most  of  the  switches,  etc.  and  as  a  result  they  will  affect  the  way  the 
tracking  task  works.  There  is  a  high  probability  that  this  will  make  the  system 
crash  outright.  Even  if  the  system  doesn’t  crash,  it  would  make  your  data 
worthless. 


Enclosed  is  a  list  of  the  symbols  being  used  for  the  Tactical  Assessment  task  and 
their  designations  as  friendly  or  hostile.  Please  study  this  list.  Also  enclosed  is  a 
practice  test  you  may  want  to  use  to  help  learn  the  symbol  codes.  Please  run  through 
it.  You  will  be  asked  to  identify  the  symbols  correctly  before  you  begin  each  session. 

The  experiment  will  require  that  you  attend  three  sessions,  each  lasting 
approximately  an  hour.  I  ask  that  you  try  to  arrange  your  schedule  so  that  the 
sessions  can  be  conducted  at  approximately  the  same  time  on  two  consecutive  days. 
Again,  thank  you  for  your  agreeing  to  participate  in  this  study. 
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Tracking  Task  Display  Format 


(T)  Ownship  bug  (fixed  to  display) 

(2)  Target  (to  be  tracked) 

(3)  Ground  representation 

(4)  Altitude  scale 


NADC-91 028-60 


Tactical  Assessment  Task  (TAT)  Display  Format 


(T)  Ownship 

(?)  Outer  Range  Ring  (As  targets  approach  they  must  be  identified) 
(?)  Target  (to  be  identified) 

(4)  'TARGET'  Prompt-  signals  that  a  target  must  be  identified 
(?)  Target  which  has  been  correctly  identified 
(?)  Target  which  will  need  to  be  identified 
(?)  Response  buttons 

(?)  Target  which  has  been  missed  or  incorrectly  identified 
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Please  use  this  short  test  to  see  how  well  you  can 
identify  each  target  symbol  as  being  either  Friendly  or  Hostile. 
Friendly  symbols  get  confirmed;  Hostile  symbols  get  designated. 
Circle  the  corresponding  'letter  for  Confirmate  (C)  or  Designate  (D). 


You  must  be  able  to  identify  all  symbols  correctly. 


When  you 

have  finished, 

check  your  responses 
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